Voice Actions on Computing Devices

ABSTRACT

A computer-implemented method includes receiving spoken input at a computing device from a user of the computing device, the spoken input including a carrier phrase and a subject to which the carrier phrase is directed, providing at least a portion of the spoken input to a server system in audio form for speech-to-text conversion by the server system, the portion including the subject to which the carrier phrase is directed, receiving from the server system instructions for automatically performing an operation on the computing device, the operation including an action defined by the carrier phrase using parameters defined by the subject, and automatically performing the operation on the computing device.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation application of U.S. application Ser.No. 12/914,965, filed on Oct. 28, 2010, and titled VOICE ACTIONS ONCOMPUTING DEVICES, which claims the benefit under 35 U.S.C. §119(e) ofU.S. Provisional Application Ser. No. 61/255,847, filed on Oct. 28,2009, entitled “Search, Navigation, and Docking with a Mobile ComputingDevice,” the entire contents of which are hereby incorporated byreference.

TECHNICAL FIELD

This document relates to actions that may be taken by or with a mobilecomputing device such as a smartphone, including by controlling anavigation application using voice inputs.

BACKGROUND

Mobile communication devices allow users to make telephone calls,receive email, browse the World Wide Web, listen to audio content, andview video content. Such devices have gotten more powerful over theyears, to the point where they can now execute various custom,downloaded applications for a variety of needs. Many of the applicationsare very sophisticated and may access server-based data automaticallywhile they are running so as to provide a rich user experience.

The number and type of sensors on smartphones has so proliferated inrecent years. Many such devices now have electronic compasses,accelerometers, GPS units, cameras, proximity sensors, and other suchsensors. These sensors can be used in a variety of manners, such as todetermine a user's location with a GPS unit, and the user's orientationwith a compass unit, and then to provide a compass-based GOOGLESTREETVIEW around the user. Other applications can provide basicturn-by-turn navigation in response to a user's provision of an addressto a device. Also, dedicated navigation units permit a user to type in adestination address and to have turn-by-turn directions provided betweenthe user's current location and the destination address.

SUMMARY

This document describes systems and techniques for conducting search(and particularly voice search) and navigation (including voice-basednavigation) on a mobile computing device, and for performing actionsassociated with docking of a mobile computing device. In certainexamples, a user can speak the name of a venue where they would like totravel, perhaps preceded by a control term, carrier phrase, or triggerterm, such as “navigate to,” and may have navigation instructionsprepared automatically so that the computing device into which theyspoke the destination name will automatically give them turn-by-turndirections (including with spoken directions, e.g., “turn right in 300feet”) from their current geographic location to the spoken venue. Thespoken input may be transmitted by the device to a remote server system,which may identify the presence of the control term, and then use thepresence of the control term in order to determine how to handle therest of the spoken input or query. For example, the control term“navigate to” may cause the system to enter a textual version of therest of the query to a local search engine, and may use the results toidentify an address for the destination venue, and then in turn maydetermine a lat/long pair or other identifier that can be used as anendpoint for navigation directions (along with the current location ofthe user and their device, determined, e.g., using GPS functionality onthe device). Such geographic location information may then be suppliedto a further component of the system as an endpoint for a route, and thefurther component (a navigation engine) may provide navigationinformation back to a user of a mobile device. Such information mayinclude data required to draw a standard route along a navigable route,to animate the path along the route as the user drives, such as with amoving dot over a map, with digital photographic images of the actualarea along that route that has been previously captured by cars or othervehicles driving the route.

Navigation applications also can provide a presentation of navigationinformation in response to natural language queries from a user that donot include a location name. For example, a user can, instead ofsubmitting the name of a point of interest, such as the name of amuseum, request to be given directions to the “museum that is currentlyholding a Picasso exhibition.” To generate endpoints for such anavigable route, a system may submit terms such as “museum currentlyholding Picasso exhibition” to a local search engine (e.g., using apublic API for submitting queries), along with an indication of theuser's current location. The search engine may then, according to itsstandard mode of operation, return various search results that are inthe relevant area and responsive to the query. The system may then parsethe search results, such as parsing landing pages that the searchresults point to, to identify geographic locations listed on the pages,such as addresses of museums. As one example, a top result may be thehome page for the Hirshhorn Museum, and that page can be parse for textthat matches a format for an address, or can be searched for a furtherlink such as “directions” or “contact us,” and the target page may thenbe parsed to obtain address information that can then be converted tolat/long coordinates. Also, once the venue name is identified from thesearch results, it may be submitted to a structured database, such as ayellow pages or similar database that correlates business or venue namesto addresses and telephone numbers.

The user's mobile device may also be programmed to listen for particularcontrol terms even if the mobile device does not perform thespeech-to-text conversion on the queries. The device may thus determinewhich server system to send the rest of the audio spoken query to, andto select the categories of meta data to send with the query. In thismanner, “listening” by a device may be only local, so that informationspoken by a user goes to a server system only when the useraffirmatively speaks a relevant control term. When the device listens inthis manner, it may also be hashing the input so as to preserve moreprivacy for the user, and may compare the hashed input to a pre-savedhashed version of the control term. In addition, the listening may beconfined to the computing device in certain circumstances, and notshared with a separate server system. Moreover, a user may be requiredto press an on-screen icon on the device (such as a microphone) beforethe device will listen for speech input.

The words that the user speaks after those control terms may be suppliedto the search engine as arguments for the relevant action that relatesto the control terms. Other control terms may also be predefined for asystem (e.g., by developers of an operating system for a device), orthird parties may provide their own control terms that will be listenedfor by the system, according to an API that has been defined for suchsubmissions. Third parties may also provide information for a syntaxrelated to their control terms that may be used by the system to parseuser input that is received along with the relevant control terms. Asone example, a media player application may register the control term“listen to,” along with a syntax that indicates that terms coming afterthe control term should be searched first, in tracks of music stored onthe device, and second in the names of musical groups on the device. Inaddition, the grammar that is used to convert the remainder of a spokeninput into text may be selected based on the control term that begins aninput.

Various navigation output techniques are also described below. Forexample, navigation may be provided visually (and with audio prompts)using standard maps and also using aerial or satellite views, where theviews may be modified slightly to make it appear as if they have beenrotated forward away from the user to generate a three-dimensionaleffect on a user display. Additionally, or alternatively, street-levelphotographs that roughly correspond to a current location geographicallyof a user may be displayed to the user. A transition may be madeautomatically between map or satellite views on the one hand, andstreet-level views on the other, such as based on a speed of the user ora proximity of the user to an end point of a route. For example, a usermay be shown street-level views when they slow down, under theassumption that they will be able to digest the street-level image databetter at slower speeds. In addition, a user may be shown street-levelimages when they get closer to a endpoint, under the assumption thatthey will be looking carefully for their destination at such a point.

In addition, the angle of view for street-level views may change as theuser approaches the end point. In particular, the angle of view may beforward down the road along most of the route, but may turn to the sideof the road as the user approaches the end point, so that the user canmore readily see buildings on the sides of the road to determine thatthey (the user) are at a right location. A view display to a user may bemade up of multiple images, such as images previously captured at thelocation where the user is now at, and may be stitched together anddistorted so as to emphasize certain areas, such as to fit a 180 degreefront panorama on the user's display, to compress the image in front ofthe user, and to not compress the images to each side of the used (whenthe user is driving slowly), or to leave the forward-facing imageuncompressed visually (when the user is driving faster).

In another navigation implementation, a mobile device may download andcache all of the street level images when a trip is first started, andmay include local code for transitioning the images as the user moves,so that if the device's connection with a network is broken, the devicecan continue to show images all the way to the destination.

In one implementation, a computer-implemented method includes receivingspoken input at a computing device from a user of the computing device,the spoken input including a carrier phrase and a subject to which thecarrier phrase is directed, providing at least a portion of the spokeninput to a server system in audio form for speech-to-text conversion bythe server system, the portion including the subject to which thecarrier phrase is directed, receiving from the server systeminstructions for automatically performing an operation on the computingdevice, the operation including an action defined by the carrier phraseusing parameters defined by the subject, and automatically performingthe operation on the computing device. The carrier phrase can comprisean instruction (such as “navigate to”) to execute a navigationapplication, and the subject can describe a geographical landmark,including naming the geographical landmark. The operation can comprisedisplaying turn-by-turn navigation in an application, wherein data forthe turn-by-turn navigation is generated by the server system for anavigation route from a current location of the computing device to alocation described by the subject. Also, the prefix can comprise aninstruction to play a media file, and the subject describes the mediafile to be played.

In some aspects, the subject includes a name of content in the mediafile, and in others, the subject comprises a search query and the prefixidentifies operations to be conducted on search results for the searchquery. The operations can be conducted on the search results by parsinga landing page corresponding to a search result for a designated type ofinformation. Also the designated type of information can comprise alocation.

In another implementation, a computer-implemented method comprisesreceiving, at a computer server system from a remote computing device,audio spoken by a user of the computing device, the audio including asubject to which a spoken carrier phrase form the user was directed;performing speech-to-text conversion on the audio by the computer serversystem; generating data for execution on the computing device to causethe computing device to automatically perform an operation identified bythe carrier phrase using the subject as a parameter of the operation;and transmitting the data to the computing device. The method canfurther comprise receiving an textual identifier of the carrier phrase,wherein the computing device identified the carrier phrase beforeproviding the audio to the computer server system. Also, the method canadditional comprise using the identifier to select a grammar forperforming the speech-to-text conversion. The carrier phrase cancomprise an instruction (such as “navigate to”) to execute a navigationapplication, and the subject describes a geographical landmark, and thesubject can describe a geographical landmark without naming thegeographical landmark.

In some aspects, the operation comprises displaying turn-by-turnnavigation in an application, wherein data for the turn-by-turnnavigation is generated by the server system for a navigation route froma current location of the computing device to a location described bythe subject. The prefix can also comprise an instruction to play a mediafile, and the subject describes the media file to be played, and theSubject can include a name of content in the media file. The subject canalternatively or in addition comprise a search query and the prefixidentifies operations to be conducted on search results for the searchquery, and the operations to be conducted on the search results cancomprise parsing a landing page corresponding to a search result for adesignated type of information. In certain implementations, thedesignated type of information comprises a location.

In yet another implementations, one or more tangible non-transientmachine-readable media are disclosed that store instructions. Theinstructions, when executed, perform actions comprising receiving spokeninput at a computing device from a user of the computing device, thespoken input including a carrier phrase and a subject to which thecarrier phrase is directed; providing at least a portion of the spokeninput to a server system in audio form for speech-to-text conversion bythe server system, the portion including the subject to which thecarrier phrase is directed; receiving from the server systeminstructions for automatically performing an operation on the computingdevice, the operation including an action defined by the carrier phraseusing parameters defined by the subject; and automatically performingthe operation on the computing device.

The details of one or more embodiments are set forth in the accompanyingdrawings and the description below. Other features and advantages willbe apparent from the description and drawings, and from the claims.

DESCRIPTION OF DRAWINGS

FIG. 1 shows a conceptual flow of information in a navigation systemthat responds to spoken natural language queries and provides streetlevel imaging for navigation.

FIG. 2A shows a music docking system and a mobile computing device inthe form of a touchscreen smartphone.

FIG. 2B shows a schematic diagram of a docked mobile computing device.

FIG. 3 shows an example car dock for a mobile computing device.

FIG. 4 is a schematic diagram of a system for providing navigationinformation to a user of a mobile device.

FIGS. 5A-5D are flow charts of navigation-related processes forperformance on a smartphone or similar computing device.

FIGS. 6A-6E show example screen shots of a smartphone-based navigationapplication.

FIG. 7 is a conceptual diagram of a system that may be used to implementthe systems and methods described in this document.

FIG. 8 is a block diagram of computing devices that may be used toimplement the systems and methods described in this document, as eithera client or as a server or plurality of servers.

Like reference symbols in the various drawings indicate like elements.

DETAILED DESCRIPTION

This document describes systems and techniques for providingnavigational assistance (e.g., driving directions), including inresponse to spoken queries, and using various graphical techniques. Thespoken input may be preceded by a carrier phrase and may includenon-traditional content, such as a phrase that does not include anaddress or name of a location or venue—e.g., “Directions to a nearbycoffee shop.” Audio of some or all of the spoken input may betransmitted to a server system that is remote from the computing deviceon which the input was received, and the server system may convert theaudio to text, generate results to a query in the spoken input, and thenfurther perform operations to provide the computing device with programcode that is not simply a search result for the input. For example, theexample query may be parsed to generate a query of “coffee shop,” andthe term “nearby” may be used as meta data to ranked results for thesearch so that closest shops rank more highly.

A landing page for the best result may then be accessed (e.g., byaccessing a DOM for the page) and location-related information such asaddresses may be identified and parsed from it. The location-relatedinformation may then be used to look up a lat/long pair, and to theninfer a location for the result that the user intended. The system maythen pass information or begin passing information for the generationand continuation of a turn-by-turn navigation application on thecomputing device, so that as the device reports in new geographiclocations, the device may be caused to update a map or other view and togenerated verbal driving directions.

The search by a central system may also generate a variety of searchresults, which may be displayed normally in the form of snippets thatpoint to landing pages, which are the actual webpages that a user may goto in response to seeing search results and clicking on one of theresults. In this example, the system may parse the landing pages, or thelanding pages may have previously been parsed, to identify an address orother location information associated with the landing page. Variousmechanisms may be used to determine if an adequate level of certaintyfor a result has been met before returning information to a user, andultimately the address information may be provided to a navigationapplication as an endpoint for a route for the user. In other examples,where there are multiple search results and corresponding locations, theuser may be presented with a list of the results, along with theiraddresses, and the user may select one of the results as an endpoint fornavigation route. Thus, for example, the provision of navigationinformation to a user may be made an user-selectable option for searchresults for which the system was able to determine a geographicaladdress, such as by placing a “navigate to this result” or similar linkwith the particular result in a list of results.

When a user determines to receive navigation directions, the particularpresentation on the device may take the form of a map on which a movingdot is displayed that represents the current location of the computingdevice, in coordination with the user's motion along a predeterminedpath between two points. A map or similar cartoonish representation maybe provided to show the route between points, and a user's progress maybe indicated such as by placing an icon at the center of the screen thatrepresents user movement with the device and vehicle.

In examples here, such presentations may be made, and it may also besupplemented with satellite views of an area and street-level views ofthe area around the user as they navigate, or an area ahead of the usera sufficient distance so that the user can be given a preview of wherethey are heading along the path. Thus, for example a user may visuallyidentify information for a trip, and a central system may identify astarting point for the trip, and then compute a path between thestarting point and the ending point, where the starting point istypically the current location of the user. The system may then downloadinformation that is needed to provide a traditional display ofnavigation information for the user (e.g., as turn-by-turn directions),and may also download to the user's device a large plurality of digitalphotographic images along the route, where the images were previouslycaptured by photographers who had driven along portions of the route,such as by the well-known GOOGLE STREETVIEW service.

Various types of information may be displayed to a user in various waysas they move along a path. For example, the user may manually choose todisplay either map information or satellite information depending on thelevel of detail and screen clutter they would like to see. The user mayalso drop the display of data down to see street-level views, eithermanually or automatically. Automatic transitions to or from street levelmay be based, for example, on a current speed of the user or proximityof the user to the starting or ending point. In particular, when a useris going relatively slowly it may be preferable to show the user astreet level view, and the same is true when the user is close to astarting or ending point. Also, a user may prefer to see a view to theside when they are close to an end point so that they can see theparticular building they are supposed to stop in front of.

Where images along a route are provided, they may be fetched from theserver system, either all at the beginning of the trip, as the computingdevice moves along a route, or in batches along the route. The imagesmay be smoothly transitions one-to-the-next as the mobile devicetraverses a path so as to give the user a sense that they are lookingout their windshield or windows (and the images should match what theuser sees around him or her in any event, in a manner similar toaugmented reality). The system may also transition between differentdisplay types—for example, starting with a map view for most of a trip,but moving to a STREETVIEW street-level image view when the user slowsdown or nears a destination.

Particular implementations of the concepts discussed here are providedwith respect to each of the accompanying figures.

FIG. 1 shows a conceptual flow of information in a navigation systemthat responds to spoken natural language queries and provides overheadand street-level imaging for navigation. In general, the system 100receives input from a user 102 of a mobile computing device 104, such asin the form of spoken query inputs. The system 100 may use a serversystem 108 that is remote from the device 104 and can interpret thespoken inputs to provide information back to the device 104 fordisplaying a turn-by-turn navigation application, such as from adetermine the current location for the user 102 to a location that isreflected (implicitly or explicitly) in the spoken query that the user102 provided.

Referring more specifically to components in the figure, the user 102may be an owner of the device 104 and may have previously logged thedevice 104 into a user account at the server system 108 (so that thedevice 104 can be identified by the server system 108 on subsequentsubmissions), through a network such as the Internet 106. The serversystem 108 may include one or more servers that may provide a variety ofservices, such as search, mapping, electronic mail, and navigationalaids.

The submission of a query from the device 104 to the server system 108is indicated by Arrow A in the figure. The submission may be in the formof a natural language spoken query that is preceded by a carrier phrase.For example, where the user wishes to receive directions to a particulargeographic location, the preceding carrier phrase may be “navigate to.”The remainder of the query may include a name of a location, such as“123 Main Street” or “Bob's Steak House,” or may include informationthat does not explicitly identify the locations, such as “the beststeakhouse in this town.”

At the server system 108, the submission may be converted by standardspeech-to-text techniques to identify the query that indicates theaction to be taken. For example, the carrier phrase “navigate to” mayinitially be identified in order to determine what actions to take inresponding to the device 104. Such action may include determining one ormore locations for the navigation, so as to be able to generate a routeand route map data to return to the device 104.

The remainder of the spoken input (after the carrier phrase or controlwords) may be used as an argument or parameter for that action. In oneexample, when an explicit identifier for a location is provided (e.g.,“Bob's Steak House”), a look-up may be performed to identify an addressfor that location, and then by extension lat/long coordinates for thelocation. Where an explicit identifier is not provided, the text for thequery may be passed to a search engine that is part of the server system108, which may search content in the search engine index 110. Forexample the search may be conducted on a corpus of information for anarea that is around the current location of the user 102. Plus, forexample, a current location of the device 104 that is received in thetransmission indicated by Arrow A may be used to identify the locationof the user 102, and the remainder of the text may be used to identifyand rank search results for that location.

In the stated example for a steakhouse, one search result may beindicated as being better than all the others because the webpage forthe associated restaurant (the landing page for the search result)indicates that it is the best steakhouse in the particular city, andother web pages may point to that web page and contain similar language.Such information may cause that result to have a higher relevance forthe query than for best steakhouses in other locales, or local web pagesthat do not mention best steakhouses.

The determination of the action to be performed in response to a spokenimport, although described here as occurring on the server system 108,may similarly occur on the device 104. At the same time, speech-to-textconversion of the remainder of the query may occur on the server system108. Thus, for example, the device 104 may have a small grammar of itemsthat it listens for, where those items include carrier phrases that havebeen assigned to the device 104 or have been added by the user 102 ofthe device 104. Other such carrier phrases may include “listen to,” fora user indication that they would like a media player on the device 104to play a song whose title is spoken by the user after the carrierphrase, “play,” to indicate that the user would like a song or othermedia file played on the device 100 for, “go to,” to indicate that theuser would like a Web browser on the device 104 to automaticallynavigate to a web page that best matches the name of whatever term theuser speaks after the carrier phrase, and other appropriate carrierphrases.

Where the device 104 identifies the carrier phrase initially, it mayprovide relevant metadata to the server system 108 based on itsdetermination of the carrier phrase. For example, where the carrierphrase is “navigate to,” the device 104 may include information thatindicates a current geographic location of the device 104 obtained fromGPS functionality on the device 104. Such metadata that may be passed inthe communication indicated by Arrow A may assist the server system 108in preparing information to return to device 104, to enable device 104to display a navigation application properly.

Referring now to the actions of the server system 108 in providingnavigation information back to device 104, when a result has beenidentified, a geographic location for that result may be determined. Asone example, a landing page for a result may include an address or acommonly labeled link (e.g., “about us,” “directions,” or “contact us”)to a page that includes an address. Such information may be parsed fromthe page, either at the time the user enters the query or at a priortime (and stored in a structured database). Alternatively, a name of aresult, such as the name of a business, may be submitted to a look-uptable that correlates venue names with locations. In this manner andothers, the server system may generate a location that corresponds to areceived query that does not include an explicit identification of alocation such as by an address or venue name.

When the server system 108 has identified a location for the query (i.e.a destination for the navigation), it may generate a route between thedevice's current location and the determined destination location forthe query. The server system 108 may then collect map tiles 114 that arestored at the server system 108 for providing to the device 104, and mayalso collect street-level digital images 116 that were previously takenat points along the route. In addition, the server system 108 may gatherother information, such as information 118 for generating annotations tobe overlaid on a display while a user is on the route. Such informationmay include geographical information for gas stations and other pointsof interest along the route, so that user-selectable pins or other iconscan be superimposed or annotated on to a display of the route, and maybe selected by the user if they would like to learn more informationabout those particular objects.

In certain implementations, a best search result for the query enteredby the user may be determined, and navigation information may beprovided to the user immediately, particularly when the confidence thatthe result is correct is high. In other implementations, a best resultmay be determined, and before the server system 108 performs otheroperations, it may return a description of that result (e.g., a name thesteakhouse that it determined to be the best in the area) to the device104 so that the user can confirm whether the system has found thedestination that the user intended. For example, if the user asks forthe best steakhouse in town, and the system selects a fast foodrestaurant, the user may be shown a selectable control to see the nextsearch result. In certain implementations, the first result may bedisplayed to the user 102 with a countdown timer of approximately 5seconds, so that if the user 102 does not affirmatively decline thefirst identified result, navigation information for that result willautomatically be delivered to the device 104 when the countdown timerexpires. As shown below for the example of device 104, when the serversystem 108 can provide relevant information, a standard turn-by-turnnavigation implementation may be displayed on device 104, and may beaccompanied by spoken directions generated on the device 104.

As discussed more fully below, various implementations may be achievedin the display of the navigation operation to a user. For example, theprovision of map tiles or other drawn items, in addition to photographsalong a route, may enable a better display for a user of the device 104.For example, map tiles or similar drawn representations may be shown atsome points along the trip, while photographs of a route may be shown atother locations. As one example, photographs may be shown as usersarrive at their destination, so that they can better compare what theysee in real-life at a street-level view with what they see in theirnavigation application from the street-level view, such as storefronts.

Also, the map tiles along a route and on a reasonable distance on eachside of the route, in addition to relevant digital images taken fromstreet level, may be pre-cached on the device at the time the userrequests that navigation directions. The items may then be servedlocally from the device 104 as the user moves along a route. In thismanner, the relevant graphical items that are needed for the navigationapplication may all be stored in a relatively small space (as comparedto storing all of the data for a large geographic area) on the device104, and the navigation application may continue to work even if theuser 102 loses connection to a network. Such a feature may beparticularly beneficial where network outages are common, or where auser is making use of a device that may have WiFi or similarcapabilities, but not 3G or 4G cellular capabilities.

Using the techniques discussed here, a user of a mobile computing devicemay easily receive user inputs (including spoken inputs) for performingvarious operations, including generating navigation directions, and maybe provided with results that are directed toward the particularoperation they requested and at the particular data that they requestedto be processed for the operation. The presentation of the data may beprovided automatically and may be presented in a way that isparticularly useful for a user of the device.

FIG. 2A shows a music docking system 250 and a mobile computing devicein the form of a touchscreen smartphone. This figure shows schematicallya system in which a wired electrical connection is made upon docking thedevice, and the electrical connection triggers other actions on thedevice to occur. In the figure, the dock 254 takes a familiar form of anappliance that can be plugged into a standard electrical outlet and thatincludes amplifiers and speakers, among other things, and is generally ahome music dock for a mobile computing device such as a music player orsmartphone.

The actions that may be triggered include switching modes or states ofthe device automatically upon docking the device or undocking thedevice, including where the mode into which the device switches dependson the type of the dock. For example, a device may take on a particulartheme or mode when it is docked into a music-playing dock like thatshown in the figure, including by forming a BLUETOOTH link with the dockfor the playing of music over speakers in the dock. When the device isplaced into a car-related dock, in contrast, the device may take on adifferent theme, including by launching a mapping or navigationapplication, and also providing audio output to a speaker on the dock orforming a BLUETOOTH connection with an audio system in the vehicle, ifthere is such a system in the vehicle.

A first device state 252A of the mobile device is shown when the deviceis physically connected to the dock 254, and a second device state 252Bis shown when the device has subsequently been undocked. A chargingmechanism is shown schematically in the figure by supply voltage 260that is provided through a physical docking connector 258 to rechargebatteries in the device and to allow the device to operate when it isdocked. The dock 254 also has a display that can show a time, or statusrelating to the dock 254, including via information that is receivedfrom the device.

FIG. 2B shows a schematic diagram of a docked mobile computing device.In general, the schematic diagram shows various components that may beprovided on the device so as to manage the switching of modes on thedevice based on the status of the device being undocked or docked toparticular types of docks.

An input manager 212 in the device manages input to the device,including touch input to a screen on the device. The input manager 212may correlate such input with applications being displayed on the deviceso as to allow a user of the device to launch and interact with suchapplications and with components of an operating system executing on thedevice. A display manager 214 coordinates with the touch manager andcontrols what items are displayed in what positions, which will be shownover the top of other items, and which will be shown as having thedevice's focus at a particular time.

A connection manager 218 manages docked and wireless connections for thedevice, such as in the manners discussed above. For example, theconnections manager 218 may infer a context for a device based on themanner in which it is docked, and may notify various applications 216that may be run on a processor of the device 200, including navigationalapplications that are discussed more fully above and below. A powermanager 208 and voltage sensor 209 cooperate to determine when a devicehas been plugged into a power source, and also to determine when certainactivities may occur on a device after it has been plugged in. Thedevice may also determine that it has been docked, and may determine adock type for the dock, using electrical or magnetic connections betweenthe device and the dock—where the presence of connections at particularlocations represent the type of dock, or where data may be transmitted(e.g., serially) over the connections so that the device can interrogatethe dock or otherwise determine the dock type. Finally, various wirelessinterfaces 220-224 may be employed by the device 200 in order tocommunicate by a variety of modes, such as BLUETOOTH, WiFi, 3G, andother cellular connections.

FIG. 3 shows an example car dock for a mobile computing device. In thisfigure, the car dock 302 includes a base 304 for mounting the car dock302 to a surface (e.g., a car dashboard, a car windshield, a desk, or awall). The base 304 may include a suction cup 306 that enables the base304 to mount to consistent surfaces. A disk 308 can be adhesivelyattached to an uneven surface (e.g., a rippled car dashboard) to providea consistent surface for the dock. Such mounting mechanisms may be usedto mount the dock 302 to a windshield of a car so that a display on thedevice is aiming toward a driver of the car, and so that the drover mayconveniently employ various applications including navigationalapplications.

The base 304 connects to the docking body 310 via a rod. A first end ofthe rod connects with the base 304 using a ball joint, while a secondend of the rod connects to the docking body 310 with another ball joint.Thus, the docking body 310 may be pivoted to face differentorientations, and may rotate between landscape and portraitorientations.

The docking body 310 includes a recessed inner portion 302 that issurrounded by a wall 312. The recessed inner portion 302 and the wall312 are shaped to correspond to a particular model of mobile computingdevice. A lower wall 312 a and a retractable top latch 316 may at leastpartially enclose a front surface of the mobile device in order toconstrain the mobile device to the dock 302. In some examples, the sidewalls 312 b may not partially enclose the front surface. In other words,the walls may not circumferentially wrap more than half-way around anarrow portion of the mobile device.

The car dock 302 may include a female USB jack 314 to receive a USB cord(not shown) that supplies power, and in some examples, data. The cardock 302 also includes speakers 318 that radiate sound from a back sideof the dock. In some examples, the speakers play audio that has beenwirelessly transmitted from a mobile device to the dock, such as spokenturn-by-turn navigation commands. A volume rocker switch 326 allows auser to adjust the volume of audio that is output by the speakers 318.The car dock 302 also includes a microphone 334. The dock microphone 334may have acoustic qualities that are better suited for speakerphoneoperation than a microphone of a mobile device that is received in thedock. Thus, the dock 302 may wirelessly transmit, to a docked device, anaudio signal that is received using the dock microphone 334.

A first pin 332 provides a positive voltage electrical signal to adocked device. A second pin 330 provides a reference voltage to thedevice. A third pin 328 may be used for serial data communicationbetween the device and the dock 302. The pins 328-432 may be provided ina same spacing and orientation as the pins 306-310 of FIG. 3. In otherwords, the pins in both devices contact the same portions of a mobilecomputing device that is moved between the docks.

A device may be programmed to recognize the presence of electrical powerwhen it is physically docked and, as a result of such recognition, tothen begin a process of determining a sock type for the dock (e.g.,music dock, car dock, etc.). Where the dock type is determined to be amusic dock, a music application may be automatically launched on thedevice, and a BLUETOOTH connection made to begin passing audio from thedevice to the dock (where the connection may be maintained even afterthe device is removed from the dock). A similar action may occur for acar dock, but alternatively or in addition, a navigation application maybe launched automatically and may display a map or other visualindication of the user's current location (as determined, e.g., from asensor unit on the device that includes GPS functionality) and mayreceive input for a destination location, in manners described above andbelow, so as to automatically begin providing to a user turn-by-turndirections or other navigational instructions.

In certain instances, the device may be programmed to automaticallyobtain or provide certain information through the dock or with a systemthat is associated with the dock, in response to being docked. Forexample, when the dock is in an automobile, the device may obtaininformation about the current status of the automobile, including thecurrent RPMs of the automobile, the gear in which the automobile isoperating, a speed of the automobile, the status of air-conditioning andentertainment systems in the automobile, and other information. Inaddition, the device may be programmed to operate as a partial orcomplete head unit for an entertainment system in the automobile (or ascreen in the car can act as a head unit for the device). For example,the automobile may be provided only with amplifiers, speakers, andassociated hardware, but not input and output mechanisms. The mobilecomputing device may serve the role of the input and output mechanisms,and may also provide for encoding and decoding of digital music andother such common operations. In this way, an owner of an automobile maybe able to update the interaction of their car stereo by updatingsoftware on the mobile computing device, and may customize the way thattheir entertainment system in their automobile interacts with them. Alsoas the user acquires newer mobile computing devices, the user'sexperience with their mobile entertainment system in their automobilemay improve. Moreover, the user may have a preferential experiencebecause the user interface that they face will not be different orsubstantially different between their interaction when they are at homeor walking around, as compared to when they are driving theirautomobile. The mobile computing device may also act as a supplementalinterface for an interface that is part of the automobile. For example,the automobile may provide for radio station selection and other commonactions, while the mobile computing device may provide for selection ofparticular song titles or other media files that are stored on themobile computing device.

As yet another implementation relating to docking, separate wired andwireless connections may be made between a mobile computing device and adock, or a device associated with a dock, when the mobile computingdevice is physically docked. The wired connection may be a powerconnection in a familiar manner, and may be used to recharge the mobiledevice while it is docked. A wired data connection could also beprovided, but that data connection will be broken if and when the mobilecomputing device is removed from the dock. Thus, as an alternative to,or in addition to, the wired data connection, a wireless data connectionmay be formed automatically when the docking occurs. Such a connectionmay be used to transfer files or other information between the mobilecomputing device and the dock or a device associated with the dock, evenafter the device has been undocked.

Thus, for example, a user may dock their portable computing device to amusic or video system, and a wireless handshaking negotiation may occurto establish a data link between the wireless computing device and thedock. The user may then choose to begin playing a media file, and themedia file may play through the dock such as when the dock is a speakersystem, or through a device associated with the dock, such as a videomonitor connected to the dock, where the dock provides for televisionplaying, recording, and similar functions, such as personal videorecorder functions. After a time period, the user may want to take themobile computing device with them but continue playing the media file orfiles that are stored on the mobile computing device. The user may thusundock the device, and the data transmission may continue over thewireless interface between the mobile computing device and the dock.With the device undocked, the user may employ the device like a remotecontrol, such as by moving throughout the user's home and changing mediafiles, adjusting volume levels, changing rooms in which the media willbe played, or other similar actions.

FIG. 4 is a block diagram of a mobile device 422 and system 420 forproviding navigation information to a user of the device 422. Ingeneral, the system 420 includes software operating on the device 422 incooperation with software at a server system 432 executing a hostedversion of a navigation application. In such an example, the device 422may interact with a user, and may transmit information for variouspieces of the processing to be performed on the server system 432, suchas speech-to-text conversion, converting search queries into geographiclocations such as in a lat/long format, and serving map tile or imagesin coordination with data that may permit a navigation application 430executing on the device 422 to interact with a user in the mannersdescribed above and below.

In the example shown, the mobile device 422 is a smartphone. In otherimplementations, the mobile device 422 can be a personal digitalassistant, a laptop computer, a net book, a camera, a wrist watch, oranother type of mobile electronic device. The mobile device 422 includesa camera and a display screen 423 for displaying text, images, andgraphics to a user, including images captured by the camera. In someimplementations, the display screen 423 is a touch screen for receivinguser input. For example, a user contacts the display screen 423 using afinger or stylus in order to select items displayed by the displayscreen 423, enter text, or control functions of the mobile device 422.The mobile device 422 further includes one or more input devices such asa track ball 424 for receiving user input. For example, the track ball424 can be used to make selections, return to a home screen, to scrollthrough multiple items in a group, or to control functions of the mobiledevice 422. As another example, the one or more input devices includes aclick wheel for scrolling through menus and text.

The mobile device 422 includes a number of modules for controllingfunctions of the mobile device 422, including modules to control thereceipt of information and triggering the providing of navigationservices to a user of the mobile device 422. The modules can beimplemented using hardware, software, or a combination of the two. Themobile device 422 includes a display controller 426, which may beresponsible for rendering content for presentation on the display screen403. The display controller 426 may receive graphic-related content froma number of sources and may determine how the content is to be providedto a user, similar to the display manager 214 in FIG. 2B. For example, anumber of different windows for various applications 442 on the mobiledevice 422 may need to be displayed, and the display controller 426 maydetermine which to display, which to hide, and what to display or hidewhen there is overlap between various graphical objects. The displaycontroller 426 can include various components to provide particularfunctionality for interacting with displayed components, which may beshared across multiple applications, and may be supplied, for example,by an operating system of the mobile device 422.

An input controller 428, which may be the same as or similar to theinput manager 212 of FIG. 2B, may be responsible for translatingcommands provided by a user of mobile device 422. For example, suchcommands may come from a keyboard, from touch screen functionality ofthe display screen 423, from trackball 424, or from other such sources,including dedicated buttons or soft buttons (e.g., buttons whosefunctions may change over time, and whose functions may be displayed onareas of the display screen 403 that are adjacent to the particularbuttons). The input controller 428 may determine, for example, in whatarea of the display commands are being received, and thus in whatapplication being shown on the display the commands are intended for. Inaddition, it may interpret input motions on the touch screen 423 into acommon format and pass those interpreted motions (e.g., short press,long press, flicks, and straight-line drags) to the appropriateapplication. The input controller 428 may also report such inputs to anevent manager (not shown) that in turn reports them to the appropriatemodules or applications. For example, a user viewing an options menudisplayed on the display screen 423 selects one of the options using oneof the track ball 424 or touch screen functionality of the mobile device422. The input controller 428 receives the input and causes the mobiledevice 422 to perform functions based on the input.

A variety of applications 442 may operate, generally via a commonmicroprocessor, on the mobile device 422. The applications 442 may takea variety of forms, such as mapping and navigation applications, e-mailand other messaging applications, image viewing and editingapplications, video capture and editing applications, web browserapplications, music and video players, and various applications runningwithin a web browser or running extensions of a web browser. In certaininstances, one of the applications, a navigation application 430, may beprogrammed to communicate information to server system 432 via network450.

A wireless interface 440 manages communication with a wireless network,which may be a data network that also carries voice communications. Thewireless interface 440 may operate in a familiar manner, such asaccording to the examples discussed below, and may provide forcommunication by the mobile device 422 with messaging services such astext messaging, e-mail, and telephone voice mail messaging. In addition,the wireless interface 440 may support downloads and uploads of contentand computer code over the wireless network. The wireless interface 440may also communicate over short-range networks, such as with otherdevices in the same room as device 422, such as when results areprovided to the device 422 and need to be forwarded automatically toanother device in the manners discussed above and below.

A camera controller 432 of the mobile device 422 receives image datafrom the camera and controls functionality of the camera. For example,the camera controller 432 can receive image data for one or more images(e.g. stationary pictures or real-time video images) from the camera andcan provide the image data to the display controller 426 and/or to oneor more of the application 442.

Still referring to FIG. 4, in accordance with some implementations, thenavigation application 430 uses a GPS Unit 438 of the mobile device 422to determine the location of the mobile device 422. For example, the GPSUnit 438 receives signals from one or more global positioningsatellites, and can use the signals to determine the current location ofthe mobile device 422. In some implementations, rather than the GPS Unit438, the mobile device 422 includes a module that determines a locationof the mobile device 422 using transmission tower triangulation oranother method of location identification. In some implementations, themobile device 422 uses location information that is determined using theGPS Unit 438 to identify geo-coded information that is associated withthe location of the mobile device 422. In such implementations, locationinformation obtained or determined by the GPS Unit 438 is provided tothe navigation application 430. In some implementations, the navigationapplication 430 uses the location information to identify geo-coded data446 stored on the mobile device 422.

The geo-coded data 446 includes information associated with particulargeographic locations. For example, geo-coded data can include buildingnames, business names and information, historical information, images,video files, and audio files associated with a particular location. Asanother example, geo-coded data associated with a location of a park mayinclude hours for the park, the name of the park, information on plantslocated within the park, information on statues located within the park,historical information about the park, and park rules (e.g. “no dogsallowed”). The geo-coded information can also include map tiles ordigital images to be displayed to a user of the device 422.

The navigation application 430 can use the current location of themobile device 422 to identify information associated with geographiclocations that are in close proximity to the location of the mobiledevice 422, such as for annotating a display of a navigation applicationwith information such as information for local businesses that a usermay want to visit. In some implementations, the geo-coded data 446 isstored on a memory of the mobile device 422, such as a hard drive, flashdrive, or SD card. In some implementations, the mobile device 422 maycontain no pre-stored geo-coded data. In some implementations, none ofthe geo-coded data 446 stored on the mobile device 422 is associatedwith locations within relative proximity to the current location of themobile device 422. The geographical information can be used in variousways, such as passing the data to the central server system 432, so thatthe central server system may identify a current location of the mobiledevice and thereby set that location as an initial location, or may knowwhich navigation to pass to the mobile device 422 as the device moves.

The device 422 utilizes a compass unit 436, or magnetometer, in someexamples, e.g., to determine a current viewing direction of a camera onthe device 422, within the horizontal plane, of the camera. In otherwords, the compass unit 436 determines a direction in which a user ofthe mobile device 422 is looking with the mobile device 420. Viewingdirection information provided by the compass unit 436 can be used ifthe device 422 passes an image to the server system 432, such as forpurposes of the submitting a query to the server system 432, or foradding the image to a collage of images at the location from multipleusers. In some implementations, the mobile device 422 further includesan accelerometer unit 434 or a gyroscope that may be further used toidentify a user's location, movement, or other such factors.

Still referring to FIG. 4, in accordance with some implementations, themobile device 422 includes user data 448. The user data 448 can includeuser preferences or other information associated with a user of themobile device 422. For example, the user data 448 can include a numberof locations that the user has visited recently so that those locationscan be suggested over others by a navigation system (and can be added toa speech-to-text grammar if the user input is verbal). The user data 448may also indicate the manner in which the user wants navigationinformation displayed. For example, the user may always want to see amap view or a satellite view, or the user may establish pre-sets so thatmaps views are displayed under certain conditions and street views aredisplayed under other conditions.

The navigation application 430, which may run in a browser or be astand-alone application, can interact with the server system 432 in avariety of manners. For example, in collecting spoken input from a user,the device 432 may provide a general application in the operating systemfor converting spoken input to text. The server system 432 may recognizea carrier phrase in the input and may use that carrier phrase to selectan application to which the input was directed, and may pass anidentifier for the application (e.g., the navigation application 430 isthe carrier phrase was “navigate to”) back to the device 423 along withthe rest of the input in textual form. The navigation application maythen pass the text back up to the server system 432 as a query that canbe analyzed by the server system 432 to identify, e.g., a target for anavigation. Alternatively, the server system may perform thetext-to-speech and determine the location information without firstpassing the text back to the device 422. The navigation application 430may then wait to receive code and other data for interacting with theuser for the navigation, such as in the manners discussed above andbelow. For example, the navigation application may receive map tiles orstreet-level images along with data specifying geographic locations forthose objects. The navigation application may then use such informationto generate an interactive navigation experience for the user of thedevice 422.

FIGS. 5A-5D are flow charts of navigation-related processes forperformance on a smartphone or similar computing device. FIG. 5A is aflow chart of a process for generating navigational information inresponse to spoken user queries. In general, the process involvesreceiving user queries that may be preceded by a carrier phrase such as“navigate to,” and followed by a name of a location or an address, oralternatively, by a query that does not include a location name oraddress.

The process begins at box 502, where a search query is received. Thequery may be received initially at a computing device, and subsequentlyat a server system. The query may include a carrier phrase or triggeringterm that identifies the type or category of task that the user wants tohave performed. For example, as indicated above, the term may be“navigate to,” “go to,” “listen to,” “play,” “pay” to indicate the nameof a second user who should receive a transfer of payment such as in anelectronic payment system, “send to” in order to automatically form amessage such as an electronic mail message (where subsequent spokenterms may identify the content of the message), and the like. The querymay also be received without a preceding carrier phrase, and may thensimply be treated as a query that was intended to be an input to astandard search engine. In such a situation, a list of search resultsmay be presented on a display of the device in a normal manner. As aresult, ordinary voice search may be integrated with user-definedactions on a device.

At box 504, the triggering word or carrier phrase in the spoken input isidentified. Such identification may occur on the mobile device or theserver system, and may occur before or after all or a portion of thespoken input is converted to text from box 506. For example, a mobiledevice may monitor for carrier phrases when a user indicatesaffirmatively that they want such listening (e.g., by pressing amicrophone icon on a display of the device), or may continuously monitorfor spoken input while looking for a particular carrier phrase. When themobile device hears a carrier phrase, it may begin saving subsequentspoken input, and when the subsequent spoken input ends, as recognizedby a pause by the speaker, a file of the subsequent spoken input, whichmay be a query from the user, may be passed to the server system. Inaddition, metadata for the file may be passed with the input, and theparticular metadata that is chosen by the mobile device to be passed,may be dependent on the particular carrier phrase that is spoken. Forexample, if the carrier phrase is “pay,” a financial account identifierfor a user of the device may be passed to the server system, so that theserver system can automatically carry out the transaction. In contrast,when the carrier phrase is “navigate to,” the additional data may be acurrent geographic location of the mobile computing device. The metadata may also be used to assist in the conversion from speech to text,such as by selecting a grammar that matches the particularspeech-to-text conversion to be performed. For example, when the carrierphrase is “pay,” words such as dollar, point, cents, and a count may beelevated in importance in a grammar. In contrast, when the carrierphrase is “navigate to,” terms such as street, avenue, ordinaldirections, names of states and towns, and other such information may beelevated in a grammar.

At box 508, one or more search results that are responsive to the querymay be identified, such as at the server system. For example, aparticular type of search may be performed based on the carrier phrasethat was submitted, such as a local search or a search of a particularcorpus such as a yellow pages, when the carrier phrase is “navigate to.”In certain implementations, a list of search results may be returned byanalyzing an index with the search engines, and the search results mightnot themselves contain location-related information. Thus, landing pagesfor the search results or other similar document related to each of thesearch results may be analyzed for geographic location-relatedinformation, such as in manners described above (box 510).

Once information that indicates a geographical location or address for asearch result is obtained, and if the search result is determined to bea result to which the user's query was directed, the server system maybegin gathering data for providing driving directions to the user. Insome embodiments, a determination that an initially-identified searchresult is the intended result, can be confirmed by transmitting to themobile device an identifier for the initially-identified result, andwaiting for user action or inaction as a confirmation that the initialresult is either right or wrong. In a particular embodiment, userinaction after a certain time period (e.g., several seconds) may beinterpreted as an indication that the result is correct, and thenavigation information may then be prepared for transfer to the mobiledevice.

At box 512, the data for automatically generating navigationalapplication display on the mobile device is transmitted. Suchinformation is described in detail above, and can include map tiles,digital photographic images, satellite overhead images, plane-viewimages, and similar displays for turn-by-turn driving the directions,along with additional information for the directions. Additionalinformation may also be included in order to direct a navigationapplication in presenting the information to a user, and in respondingto input from the user and changes in the location of the mobile device.Particular manners in which such information may be displayed to a user,and may interact with a user, are described in more detail above andbelow.

FIG. 5B is a flow chart of a process for generating navigationalinformation that includes street-level imaging. In general, the processmay include various of the steps discussed in FIG. 5A, but may provideparticular user interactions with respect to street-level digitalphotographic images in particular areas along the route of a navigation.

The process starts at box 520, where data is transmitted for determininga start and end point for a trip relating to a mobile computing device.The data may be transmitted from the mobile device to a server, or maybe generated by a subsystem in a larger server system and transmitted toanother sub-system.

At box 522, the process receives navigational information that defines aroute and images along a route. For example, a mobile device may operateaccording to the process described in FIG. 5A to receive street-levelimages along a route determined between the start and at the end. Thereceived images may arrive in a batch of images along an entire route,and the mobile device may store the images for later display as the usermoves along the route. Alternatively, the mobile device may call foradditional images as the user moves along the route.

At box 520, the process senses motion of the mobile device. Such motionmay be determined in familiar ways by using a GPS module on the device,such as in manners commonly used to show moving on a map of a mappingapplication or navigation service. When such motion is sensed, imagesdisplayed along the route may be updated for the display, at box 526, incoordination of movement of the user along the route. For example, if auser is looking at a map view of a route and is moving south, new tilesmay need to be obtained for display at the bottom of a display screen asthe screen pans with the user's movement. Alternatively, as a user movesforward on a route, street-level photographs may need to be changed tomatch photographs taken around the user's current location on a route.The changing of such photographs or images may be blended rather thansudden, so as to give the user more of a sense of motion along the routethat matches what they are experiencing out the windows of theirvehicle.

At box 528, a viewing style is changed during the trip. The viewingstyle reflects an angle from which the route is being displayed to theuser, or a type of graphical object that is being used to display theroute. For example, map displays are typically shown from directlyabove, as are satellite displays. Also, some displays use drawn objects(maps or most navigation displays) while others use photographed images(e.g., satellite and street-level displays taken from all angles from avehicle that previously traveled a portion of the route).

As described above, the changes in viewing style may be responsive tovarious events. In one implementation, the event may be a change in thespeed of the vehicle in which the navigational directions are beingdisplayed. For example, if the vehicle speed falls below a threshold, adisplay style may change from a display of drawn figures or overheadfigures, to a display of street-level figures, whether drawn orphotographic images. Such a change may occur under the assumption that auser is more interested in seeing the faces of buildings when they aremoving slowly, whereas they are more interested in seeing a map whenthey are moving quickly. A change in display style may also occur inresponse to a location of a user, and in particular, the presence of theuser at or near a particular waypoint along a route or an endpoint onthe route. Thus, for example, if an important venue is located along aroute, such as a sports stadium, the view a user sees may be augmentedto show photographic street-level images of the stadium to the user asthe user approaches the stadium. Alternatively, when a user is in apre-determined distance of the end of a route, street-level photographsmay be shown, such as within a quarter-mile of an endpoint, under anassumption that the user will be looking for a particular venue thatthey are supposed to visit, and that they will want to see storefrontsor other identifying indications of the target location.

FIG. 5C is a flow chart of a process for generating navigationalinformation that can be used by a client device. In general, the processis similar to and can incorporate steps from the processes discussedwith respect to FIG. 5A and FIG. 5B, but is focused on the ability toobtain drawn or photographic images for a route from a server systembefore such images are needed by a mobile device, so that the images canbe displayed even if the mobile device connection to the server systemis broken during a trip.

The process begins at box 540, where a request is received thatidentifies geographic locations for a navigational trip (e.g., start andend points). For example, a server system may receive a query or otherrequest such as that discussed above with respect to FIG. 5A. At box544, the process determines a route for the trip. For example, a startpoint for the trip may be determined by a current location of therequesting device, and an endpoint for the trip may be determined bymechanisms as described above. A route for the trip may be determinedusing various techniques for identifying routes, and may take intoaccount such factors as expected roadway speeds (where freeways move atfaster speeds than local streets), and current traffic conditions, so asto provide an easy and fast route between the two points.

At box 546, navigation data for points along the route is generated. Forexample, particular geographic points may be used in a system toidentify data correlated to those points. Such data may include data forvenues such as businesses that indicated a desire to have theirinformation displayed on a map or navigational application. Theinformation may also include images, including drawn (e.g., map tiles)or photographic representations of a geography. Each such piece ofinformation may be correlated to a particular geographic point for easyrecovery of the information. Thus, for instance, a route may have pointsset at intervals along the route, such as every 10 feet, and each suchpoint may be queried by the process here to collect the images thatcorrespond to each of the points along the route.

At box 548, photographic images that previously were taken along theroute may be fetched and may be delivered to the mobile device alongwith associated metadata for generating a navigational turn-by-turndisplay for the entire route on the device, such as on a navigationalapplication executing on the device. At box 550, the navigation data andthe pre-captured images are transmitted to the mobile device. Suchtransmission may occur before any motion of the device along the routehas been identified by the server system. Thus, the transfer of all theimages, which may cover an entire route or a substantial portion of theroute, may be the last transmission that is needed from the serversystem to the mobile device in order to carry out the navigational jobat hand. In this manner, the process just described may provide aconvenient and portable mechanism by which to receive navigationalinformation, and may be employed by users who cannot afford or do nototherwise have reliable long-range data connections, such as users whorely on WiFi access points for network connections.

FIG. 5D is a flow chart of a process for identifying actions forperformance on a computing device using spoken carrier phrases. Ingeneral, the process involves receiving spoken inputs, identifyingcontrol terms or carrier phrases in those inputs, using the carrierphrases to select a particular action to perform on subsequent spokeninputs (e.g., words that are spoken immediately after a carrier phrase),converting the subsequent inputs to text, and performing the selectedaction or actions on the subsequent inputs.

The process begins at box 560, where a spoken prefix or prefixes areidentified for a speech recognition system. The prefixes may be in alist of reserved terms on a computing device, where the reserved termsare designed to reflect particular requests for particular actions by auser. Certain such terms are discussed above, and are preferablyselected to be terms that a user would speak only when they want toinvoke a particular action, and not in common conversation in othercontexts. Once the system is provided with the reserved terms, it canperform particular defined action when it hears those terms.

At box 560, the system begins monitoring voice input to determine whenthe user speaks one of the carrier phrases on the computing device. Suchmonitoring may be performed only upon a explicit indication form a userthat he or she would like to speak an input, or the monitoring may bemore continuous. Where the monitoring is more continuous, variousmechanisms may be used to protect personal information that may begathered. For example, all monitoring may occur only on the user's localdevice, and information that is monitored may be stored only for theshort time needed to determine whether a carrier phrase was spoken.Also, all monitored information may be immediately hashed, and thehashed information may be compared to hashed versions of carrierphrases, so that the actual input from the user cannot be determined,except to the extent that it can be known that the input matched acarrier phrase.

Where a prefix in the form of a carrier phrases is identified, theprocess may then store one or more terms that are spoken after theprefix or carrier phrase. For example, the device may begin recording assoon as a carrier phrase is identified, and may continue recording untila pause of a predetermined duration is made by the user.

At box 566, the spoken prefix is correlated with an action that may beperformed on the computing device. For example, the action may includethe playing of a song, the paying of an amount of money to another user,the generation of navigation or map information, and other examplesdiscussed above.

At box 510, the subsequent terms spoken by the user may be identifiedfor performing of the action on them. Such identification may includeperforming speech-to-text conversion on the stored spoken terms thatoccurred at box 564. At box 570, the process causes actions to beperformed on the computing device, to apply the identified action to theidentified subject (which is represented by the subsequently spokenwords). For example, where the action is “play” and the subject as isLynyrd Skynyrd, the action may be the playing of a song stored on amobile device or a list of songs that are identified as being sung bythe band Lynyrd Skynyrd. Other operations, where the carrier phrase actsas a verb for an action, and subsequent spoken input by a user acts as asubject for the action, may also be subsequently performed in responseto other spoken imports from a user. Generally, the verb identifies anapplication that should be executed for the input. The subjectrepresents an object or other item in the system that should be acted onby the action. Examples of such subjects include geographic locations,artist or song names, payment recipients, electronic mail messagerecipients, and other similar items.

In this manner, a user may be allowed to easily speak a shorthand for anaction into their mobile device, followed by a parameter for the action.Carrier phrases may be selected in a manner that is intuitive for users,so that users may readily control their mobile devices was spokenimports and thus enable simple hands-free operation of their devices. Asa result, users of such devices may increase their use of the devicesand may enjoy their devices more.

FIGS. 6A-6E show example screen shots of a smartphone-based navigationapplication. In general, the images show particular example userdisplays that may be generated by the systems discussed here.

FIG. 6A is a display of a map having a 3D effect, with a route along themiddle of the map, on a device 602. Turn-by-turn directions are shown ina familiar manner at the top of the display, and an estimated time toarrival is shown in the bottom corner along with a dot that is coloredto express the upcoming traffic situation (green, yellow, and red). Pinsfor various facilities are also shown along the route. A similar displayis shown to the right, though in landscape format.

The route for the navigation may have been determined using thetechniques discussed above. The information displayed on the device 602may be provided to a navigation application on the device 602 by variousserver systems and may be transmitted to the device 602 from a singleinterface server (e.g., a web server) or from multiple differentservers. For example, the background image may be served by a particularserver sub-system, while the data for generating the annotations (e.g.,pins for restaurants and other venues) may be generated by a separatesub-system. The navigation application may operate to combine suchinformation, to show a superimposed line for the path of the trip, andto update the display depending on the current geographic location ofthe device 602.

FIG. 6B is similar FIG. 6A, and shows a navigation display in alandscape orientation on a device 604. A header indicates visually to auser that they need to turn right in 1.4 miles, and such instructionsmay be followed up with verbal, spoken announcements in a familiarmanner as the turn point nears. The header also shows prominently thename of the street onto which the turn is to occur. Also, an estimatedtime until arrival is shown in a lower left corner of the display.

FIG. 6C shows an aerial or satellite view at a slight angle for a 3Deffect. Again, a line (superimposed with arrows) shows the route, thedisplay indicates that a left turn will be needed in 200 feet onto Oakstreet, the ETA is 21 minutes, and a green dot indicates that comingtraffic is light.

FIGS. 6D and 6E show street level views from a navigation application ona mobile device 608. In the left view, the user is looking forward downa street, while in the right hand image, the point of view has rotatedto the side somewhat because the destination restaurant is to be to thatside of the user. Thus, if the navigation ends on this image, the drivercan know that they are supposed to go to the facility with outdoordining and a striped awning. The name of the destination venue is alsodisplayed to confirm for the user that they are at the right place.

In other instances, the display may show a combination of forward andside images for a more immersive effect. In certain implementations, oneor more of the areas of a stitched-together panorama may be distorteddepending on the area that the device 610 has determined to be mostimportant. For example, as a user gets closer to a destination, a mapview may change to a “surround” street-level view where the images tothe side are visually compressed so that they can be sensed but not seenclearly. As the user gets closer to her destination or as she slows down(as determined using GPS function in the device), the front part of thesurround image may be compressed and the side images expanded. Such atransition may be sudden (e.g., switching from one mode to the otherwhen the speed falls below 20 miles per hour) or gradually, e.g.,changing the focus of the surround image in a linear manner as the userapproaches the end of a trip.

In addition, although the relatively small screen size shown in theseimages is small and thus limits what can be displayed with thenavigation application, other implementations may use touch-screentablet computers and may thus fit more information on such largerscreens. For example, an animated overhead map view may be displayed onone side of a screen, while street-level views and other information maybe displayed on the other half simultaneously. Other variousarrangements of a display may also be implemented, and additionalinformation may also be displayed on a device.

Referring now to FIG. 7, a conceptual diagram of a system that may beused to implement the systems and methods described in this document isillustrated. In the system, mobile computing device 710 can wirelesslycommunicate with base station 740, which can provide the mobilecomputing device wireless access to numerous hosted services 760 througha network 750.

In this illustration, the mobile computing device 710 is depicted as ahandheld mobile telephone (e.g., a smartphone, or application telephone)that includes a touchscreen display device 712 for presenting content toa user of the mobile computing device 710 and receiving touch-based userinputs. Other visual, auditory, and tactile output components may alsobe provided (e.g., LED lights, a speaker for providing tonal,voice-generated, or recorded output, or vibrating mechanisms for tactileoutput), as may various different input components (e.g., keyboard 714,physical buttons, trackballs, accelerometers, gyroscopes, andmagnetometers).

Example visual output mechanism in the form of display device 712 maytake the form of a 3.7 or 4.3 inch LED or AMOLED display with resistiveor capacitive touch capabilities, for displaying video, graphics,images, and text, and coordinating user touch inputs locationally withthe displayed information so that user contact above a displayed itemmay be associated with the item by the device 710. The mobile computingdevice 710 may take alternative forms also, including as a laptopcomputer, a tablet or slate computer, a personal digital assistant, anembedded system (e.g., a car navigation system), a desktop personalcomputer, or a computerized workstation.

An example mechanism for receiving user-input includes keyboard 714,which may be a full qwerty keyboard or a traditional keypad thatincludes keys for the digits ‘0-9’, ‘#’, and The keyboard 714 receivesinput when a user physically contacts or depresses a keyboard key. Usermanipulation of a trackball 716 or interaction with a trackpad enablesthe user to supply directional and rate of rotation information to themobile computing device 710 (e.g., to manipulate a position of a cursoron the display device 712).

The mobile computing device 710 may be able to determine a position ofphysical contact with the touchscreen display device 712 (e.g., aposition of contact by a finger or a stylus). Using the touchscreen 712,various “virtual” input mechanisms may be produced, where a userinteracts with a graphical user interface element depicted on thetouchscreen 712 by contacting the graphical user interface element. Anexample of a “virtual” input mechanism is a “software keyboard,” where akeyboard is displayed on the touchscreen and a user selects keys bypressing a region of the touchscreen 712 that corresponds to each key.

The mobile computing device 710 may include mechanical or touchsensitive buttons 718 a-d. Additionally, the mobile computing device mayinclude buttons for adjusting volume output by the one or more speakers720, and a button for turning the mobile computing device on or off. Amicrophone 722 allows the mobile computing device 710 to convert audiblesounds into an electrical signal that may be digitally encoded andstored in computer-readable memory, or transmitted to another computingdevice. The mobile computing device 710 may also include a digitalcompass, an accelerometer, proximity sensors, and ambient light sensors.

An operating system may provide an interface between the mobilecomputing device's hardware (e.g., the input/output mechanisms and aprocessor executing instructions retrieved from computer-readablemedium) and software. Example operating systems include the ANDROIDmobile device platform; APPLE IPHONE/MAC OS X operating systems;MICROSOFT WINDOWS 7/WINDOWS MOBILE operating systems; SYMBIAN operatingsystem; RIM BLACKBERRY operating system; PALM WEB operating system; avariety of UNIX-flavored operating systems; or a proprietary operatingsystem for computerized devices. The operating system may provide aplatform for the execution of application programs that facilitateinteraction between the computing device and a user.

The mobile computing device 710 may present a graphical user interfacewith the touchscreen 712. A graphical user interface is a collection ofone or more graphical interface elements and may be static (e.g., thedisplay appears to remain the same over a period of time), or may bedynamic (e.g., the graphical user interface includes graphical interfaceelements that animate without user input).

A graphical interface element may be text, lines, shapes, images, orcombinations thereof. For example, a graphical interface element may bean icon that is displayed on the desktop and the icon's associated text.In some examples, a graphical interface element is selectable withuser-input. For example, a user may select a graphical interface elementby pressing a region of the touchscreen that corresponds to a display ofthe graphical interface element. In some examples, the user maymanipulate a trackball to highlight a single graphical interface elementas having focus. User-selection of a graphical interface element mayinvoke a pre-defined action by the mobile computing device. In someexamples, selectable graphical interface elements further oralternatively correspond to a button on the keyboard 704. User-selectionof the button may invoke the pre-defined action.

In some examples, the operating system provides a “desktop” userinterface that is displayed upon turning on the mobile computing device710, activating the mobile computing device 710 from a sleep state, upon“unlocking” the mobile computing device 710, or upon receivinguser-selection of the “home” button 718 c. The desktop graphicalinterface may display several icons that, when selected with user-input,invoke corresponding application programs. An invoked applicationprogram may present a graphical interface that replaces the desktopgraphical interface until the application program terminates or ishidden from view.

User-input may manipulate a sequence of mobile computing device 710operations. For example, a single-action user input (e.g., a single tapof the touchscreen, swipe across the touchscreen, contact with a button,or combination of these at a same time) may invoke an operation thatchanges a display of the user interface. Without the user-input, theuser interface may not have changed at a particular time. For example, amulti-touch user input with the touchscreen 712 may invoke a mappingapplication to “zoom-in” on a location, even though the mappingapplication may have by default zoomed-in after several seconds.

The desktop graphical interface can also display “widgets.” A widget isone or more graphical interface elements that are associated with anapplication program that has been executed, and that display on thedesktop content controlled by the executing application program. Awidget's application program may start with the mobile telephone.Further, a widget may not take focus of the full display. Instead, awidget may only “own” a small portion of the desktop, displaying contentand receiving touchscreen user-input within the portion of the desktop.

The mobile computing device 710 may include one or morelocation-identification mechanisms. A location-identification mechanismmay include a collection of hardware and software that provides theoperating system and application programs an estimate of the mobiletelephone's geographical position. A location-identification mechanismmay employ satellite-based positioning techniques, base stationtransmitting antenna identification, multiple base stationtriangulation, internet access point IP location determinations,inferential identification of a user's position based on search enginequeries, and user-supplied identification of location (e.g., by“checking in” to a location).

The mobile computing device 710 may include other application modulesand hardware. A call handling unit may receive an indication of anincoming telephone call and provide a user capabilities to answer theincoming telephone call. A media player may allow a user to listen tomusic or play movies that are stored in local memory of the mobilecomputing device 710. The mobile telephone 710 may include a digitalcamera sensor, and corresponding image and video capture and editingsoftware. An internet browser may enable the user to view content from aweb page by typing in an addresses corresponding to the web page orselecting a link to the web page.

The mobile computing device 710 may include an antenna to wirelesslycommunicate information with the base station 740. The base station 740may be one of many base stations in a collection of base stations (e.g.,a mobile telephone cellular network) that enables the mobile computingdevice 710 to maintain communication with a network 750 as the mobilecomputing device is geographically moved. The computing device 710 mayalternatively or additionally communicate with the network 750 through aWi-Fi router or a wired connection (e.g., Ethernet, USB, or FIREWIRE).The computing device 710 may also wirelessly communicate with othercomputing devices using BLUETOOTH protocols, or may employ an ad-hocwireless network.

A service provider that operates the network of base stations mayconnect the mobile computing device 710 to the network 750 to enablecommunication between the mobile computing device 710 and othercomputerized devices that provide services 760. Although the services760 may be provided over different networks (e.g., the serviceprovider's internal network, the Public Switched Telephone Network, andthe Internet), network 750 is illustrated as a single network. Theservice provider may operate a server system 752 that routes informationpackets and voice data between the mobile computing device 710 andcomputing devices associated with the services 760.

The network 750 may connect the mobile computing device 710 to thePublic Switched Telephone Network (PSTN) 762 in order to establish voiceor fax communication between the mobile computing device 710 and anothercomputing device. For example, the service provider server system 752may receive an indication from the PSTN 762 of an incoming call for themobile computing device 710. Conversely, the mobile computing device 710may send a communication to the service provider server system 752initiating a telephone call with a telephone number that is associatedwith a device accessible through the PSTN 762.

The network 750 may connect the mobile computing device 710 with a Voiceover Internet Protocol (VoIP) service 764 that routes voicecommunications over an IP network, as opposed to the PSTN. For example,a user of the mobile computing device 710 may invoke a VoIP applicationand initiate a call using the program. The service provider serversystem 752 may forward voice data from the call to a VoIP service, whichmay route the call over the internet to a corresponding computingdevice, potentially using the PSTN for a final leg of the connection.

An application store 766 may provide a user of the mobile computingdevice 710 the ability to browse a list of remotely stored applicationprograms that the user may download over the network 750 and install onthe mobile computing device 710. The application store 766 may serve asa repository of applications developed by third-party applicationdevelopers. An application program that is installed on the mobilecomputing device 710 may be able to communicate over the network 750with server systems that are designated for the application program. Forexample, a VoIP application program may be downloaded from theApplication Store 766, enabling the user to communicate with the VoIPservice 764.

The mobile computing device 710 may access content on the internet 768through network 750. For example, a user of the mobile computing device710 may invoke a web browser application that requests data from remotecomputing devices that are accessible at designated universal resourcelocations. In various examples, some of the services 760 are accessibleover the internet.

The mobile computing device may communicate with a personal computer770. For example, the personal computer 770 may be the home computer fora user of the mobile computing device 710. Thus, the user may be able tostream media from his personal computer 770. The user may also view thefile structure of his personal computer 770, and transmit selecteddocuments between the computerized devices.

A voice recognition service 772 may receive voice communication datarecorded with the mobile computing device's microphone 722, andtranslate the voice communication into corresponding textual data. Insome examples, the translated text is provided to a search engine as aweb query, and responsive search engine search results are transmittedto the mobile computing device 710.

The mobile computing device 710 may communicate with a social network774. The social network may include numerous members, some of which haveagreed to be related as acquaintances. Application programs on themobile computing device 710 may access the social network 774 toretrieve information based on the acquaintances of the user of themobile computing device. For example, an “address book” applicationprogram may retrieve telephone numbers for the user's acquaintances. Invarious examples, content may be delivered to the mobile computingdevice 710 based on social network distances from the user to othermembers. For example, advertisement and news article content may beselected for the user based on a level of interaction with such contentby members that are “close” to the user (e.g., members that are“friends” or “friends of friends”).

The mobile computing device 710 may access a personal set of contacts776 through network 750. Each contact may identify an individual andinclude information about that individual (e.g., a phone number, anemail address, and a birthday). Because the set of contacts is hostedremotely to the mobile computing device 710, the user may access andmaintain the contacts 776 across several devices as a common set ofcontacts.

The mobile computing device 710 may access cloud-based applicationprograms 778. Cloud-computing provides application programs (e.g., aword processor or an email program) that are hosted remotely from themobile computing device 710, and may be accessed by the device 710 usinga web browser or a dedicated program. Example cloud-based applicationprograms include GOOGLE DOCS word processor and spreadsheet service,GOOGLE GMAIL webmail service, and PICASA picture manager.

Mapping service 780 can provide the mobile computing device 710 withstreet maps, route planning information, and satellite images. Anexample mapping service is GOOGLE MAPS. The mapping service 780 may alsoreceive queries and return location-specific results. For example, themobile computing device 710 may send an estimated location of the mobilecomputing device and a user-entered query for “pizza places” to themapping service 780. The mapping service 780 may return a street mapwith “markers” superimposed on the map that identify geographicallocations of nearby “pizza places.”

Turn-by-turn service 782 may provide the mobile computing device 710with turn-by-turn directions to a user-supplied destination. Forexample, the turn-by-turn service 782 may stream to device 710 astreet-level view of an estimated location of the device, along withdata for providing audio commands and superimposing arrows that direct auser of the device 710 to the destination.

Various forms of streaming media 784 may be requested by the mobilecomputing device 710. For example, computing device 710 may request astream for a pre-recorded video file, a live television program, or alive radio program. Example services that provide streaming mediainclude YOUTUBE and PANDORA.

A micro-blogging service 786 may receive from the mobile computingdevice 710 a user-input post that does not identify recipients of thepost. The micro-blogging service 786 may disseminate the post to othermembers of the micro-blogging service 786 that agreed to subscribe tothe user.

A search engine 788 may receive user-entered textual or verbal queriesfrom the mobile computing device 710, determine a set ofinternet-accessible documents that are responsive to the query, andprovide to the device 710 information to display a list of searchresults for the responsive documents. In examples where a verbal queryis received, the voice recognition service 772 may translate thereceived audio into a textual query that is sent to the search engine.

These and other services may be implemented in a server system 790. Aserver system may be a combination of hardware and software thatprovides a service or a set of services. For example, a set ofphysically separate and networked computerized devices may operatetogether as a logical server system unit to handle the operationsnecessary to offer a service to hundreds of individual computingdevices.

In various implementations, operations that are performed “in response”to another operation (e.g., a determination or an identification) arenot performed if the prior operation is unsuccessful (e.g., if thedetermination was not performed). Features in this document that aredescribed with conditional language may describe implementations thatare optional. In some examples, “transmitting” from a first device to asecond device includes the first device placing data into a network forreceipt by the second device, but may not include the second devicereceiving the data. Conversely, “receiving” from a first device mayinclude receiving the data from a network, but may not include the firstdevice transmitting the data.

FIG. 8 is a block diagram of computing devices 800, 850 that may be usedto implement the systems and methods described in this document, aseither a client or as a server or plurality of servers. Computing device800 is intended to represent various forms of digital computers, such aslaptops, desktops, workstations, personal digital assistants, servers,blade servers, mainframes, and other appropriate computers. Computingdevice 850 is intended to represent various forms of mobile devices,such as personal digital assistants, cellular telephones, smartphones,and other similar computing devices. Additionally computing device 800or 850 can include Universal Serial Bus (USB) flash drives. The USBflash drives may store operating systems and other applications. The USBflash drives can include input/output components, such as a wirelesstransmitter or USB connector that may be inserted into a USB port ofanother computing device. The components shown here, their connectionsand relationships, and their functions, are meant to be exemplary only,and are not meant to limit implementations described and/or claimed inthis document.

Computing device 800 includes a processor 802, memory 804, a storagedevice 806, a high-speed interface 808 connecting to memory 804 andhigh-speed expansion ports 810, and a low speed interface 812 connectingto low speed bus 814 and storage device 806. Each of the components 802,804, 806, 808, 810, and 812, are interconnected using various busses,and may be mounted on a common motherboard or in other manners asappropriate. The processor 802 can process instructions for executionwithin the computing device 800, including instructions stored in thememory 804 or on the storage device 806 to display graphical informationfor a GUI on an external input/output device, such as display 816coupled to high speed interface 808. In other implementations, multipleprocessors and/or multiple buses may be used, as appropriate, along withmultiple memories and types of memory. Also, multiple computing devices800 may be connected, with each device providing portions of thenecessary operations (e.g., as a server bank, a group of blade servers,or a multi-processor system).

The memory 804 stores information within the computing device 800. Inone implementation, the memory 804 is a volatile memory unit or units.In another implementation, the memory 804 is a non-volatile memory unitor units. The memory 804 may also be another form of computer-readablemedium, such as a magnetic or optical disk.

The storage device 806 is capable of providing mass storage for thecomputing device 800. In one implementation, the storage device 806 maybe or contain a computer-readable medium, such as a floppy disk device,a hard disk device, an optical disk device, or a tape device, a flashmemory or other similar solid state memory device, or an array ofdevices, including devices in a storage area network or otherconfigurations. A computer program product can be tangibly embodied inan information carrier. The computer program product may also containinstructions that, when executed, perform one or more methods, such asthose described above. The information carrier is a computer- ormachine-readable medium, such as the memory 804, the storage device 806,or memory on processor 802.

The high speed controller 808 manages bandwidth-intensive operations forthe computing device 800, while the low speed controller 812 manageslower bandwidth-intensive operations. Such allocation of functions isexemplary only. In one implementation, the high-speed controller 808 iscoupled to memory 804, display 816 (e.g., through a graphics processoror accelerator), and to high-speed expansion ports 810, which may acceptvarious expansion cards (not shown). In the implementation, low-speedcontroller 812 is coupled to storage device 806 and low-speed expansionport 814. The low-speed expansion port, which may include variouscommunication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet)may be coupled to one or more input/output devices, such as a keyboard,a pointing device, a scanner, or a networking device such as a switch orrouter, e.g., through a network adapter.

The computing device 800 may be implemented in a number of differentforms, as shown in the figure. For example, it may be implemented as astandard server 820, or multiple times in a group of such servers. Itmay also be implemented as part of a rack server system 824. Inaddition, it may be implemented in a personal computer such as a laptopcomputer 822. Alternatively, components from computing device 800 may becombined with other components in a mobile device (not shown), such asdevice 850. Each of such devices may contain one or more of computingdevice 800, 850, and an entire system may be made up of multiplecomputing devices 800, 850 communicating with each other.

Computing device 850 includes a processor 852, memory 864, aninput/output device such as a display 854, a communication interface866, and a transceiver 868, among other components. The device 850 mayalso be provided with a storage device, such as a microdrive or otherdevice, to provide additional storage. Each of the components 850, 852,864, 854, 866, and 868, are interconnected using various buses, andseveral of the components may be mounted on a common motherboard or inother manners as appropriate.

The processor 852 can execute instructions within the computing device850, including instructions stored in the memory 864. The processor maybe implemented as a chipset of chips that include separate and multipleanalog and digital processors. Additionally, the processor may beimplemented using any of a number of architectures. For example, theprocessor 410 may be a CISC (Complex Instruction Set Computers)processor, a RISC (Reduced Instruction Set Computer) processor, or aMISC (Minimal Instruction Set Computer) processor. The processor mayprovide, for example, for coordination of the other components of thedevice 850, such as control of user interfaces, applications run bydevice 850, and wireless communication by device 850.

Processor 852 may communicate with a user through control interface 858and display interface 856 coupled to a display 854. The display 854 maybe, for example, a TFT (Thin-Film-Transistor Liquid Crystal Display)display or an OLED (Organic Light Emitting Diode) display, or otherappropriate display technology. The display interface 856 may compriseappropriate circuitry for driving the display 854 to present graphicaland other information to a user. The control interface 858 may receivecommands from a user and convert them for submission to the processor852. In addition, an external interface 862 may be provide incommunication with processor 852, so as to enable near areacommunication of device 850 with other devices. External interface 862may provide, for example, for wired communication in someimplementations, or for wireless communication in other implementations,and multiple interfaces may also be used.

The memory 864 stores information within the computing device 850. Thememory 864 can be implemented as one or more of a computer-readablemedium or media, a volatile memory unit or units, or a non-volatilememory unit or units. Expansion memory 874 may also be provided andconnected to device 850 through expansion interface 872, which mayinclude, for example, a SIMM (Single In Line Memory Module) cardinterface. Such expansion memory 874 may provide extra storage space fordevice 850, or may also store applications or other information fordevice 850. Specifically, expansion memory 874 may include instructionsto carry out or supplement the processes described above, and mayinclude secure information also. Thus, for example, expansion memory 874may be provide as a security module for device 850, and may beprogrammed with instructions that permit secure use of device 850. Inaddition, secure applications may be provided via the SIMM cards, alongwith additional information, such as placing identifying information onthe SIMM card in a non-hackable manner.

The memory may include, for example, flash memory and/or NVRAM memory,as discussed below. In one implementation, a computer program product istangibly embodied in an information carrier. The computer programproduct contains instructions that, when executed, perform one or moremethods, such as those described above. The information carrier is acomputer- or machine-readable medium, such as the memory 864, expansionmemory 874, or memory on processor 852 that may be received, forexample, over transceiver 868 or external interface 862.

Device 850 may communicate wirelessly through communication interface866, which may include digital signal processing circuitry wherenecessary. Communication interface 866 may provide for communicationsunder various modes or protocols, such as GSM voice calls, SMS, EMS, orMMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS, among others.Such communication may occur, for example, through radio-frequencytransceiver 868. In addition, short-range communication may occur, suchas using a Bluetooth, WiFi, or other such transceiver (not shown). Inaddition, GPS (Global Positioning System) receiver module 870 mayprovide additional navigation- and location-related wireless data todevice 850, which may be used as appropriate by applications running ondevice 850.

Device 850 may also communicate audibly using audio codec 860, which mayreceive spoken information from a user and convert it to usable digitalinformation. Audio codec 860 may likewise generate audible sound for auser, such as through a speaker, e.g., in a handset of device 850. Suchsound may include sound from voice telephone calls, may include recordedsound (e.g., voice messages, music files, etc.) and may also includesound generated by applications operating on device 850.

The computing device 850 may be implemented in a number of differentforms, as shown in the figure. For example, it may be implemented as acellular telephone 880. It may also be implemented as part of asmartphone 882, personal digital assistant, or other similar mobiledevice.

Various implementations of the systems and techniques described here canbe realized in digital electronic circuitry, integrated circuitry,specially designed ASICs (application specific integrated circuits),computer hardware, firmware, software, and/or combinations thereof.These various implementations can include implementation in one or morecomputer programs that are executable and/or interpretable on aprogrammable system including at least one programmable processor, whichmay be special or general purpose, coupled to receive data andinstructions from, and to transmit data and instructions to, a storagesystem, at least one input device, and at least one output device.

These computer programs (also known as programs, software, softwareapplications or code) include machine instructions for a programmableprocessor, and can be implemented in a high-level procedural and/orobject-oriented programming language, and/or in assembly/machinelanguage. As used herein, the terms “machine-readable medium”“computer-readable medium” refers to any computer program product,apparatus and/or device (e.g., magnetic discs, optical disks, memory,Programmable Logic Devices (PLDs)) used to provide machine instructionsand/or data to a programmable processor, including a machine-readablemedium that receives machine instructions as a machine-readable signal.The term “machine-readable signal” refers to any signal used to providemachine instructions and/or data to a programmable processor.

To provide for interaction with a user, the systems and techniquesdescribed here can be implemented on a computer having a display device(e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor)for displaying information to the user and a keyboard and a pointingdevice (e.g., a mouse or a trackball) by which the user can provideinput to the computer. Other kinds of devices can be used to provide forinteraction with a user as well; for example, feedback provided to theuser can be any form of sensory feedback (e.g., visual feedback,auditory feedback, or tactile feedback); and input from the user can bereceived in any form, including acoustic, speech, or tactile input.

The systems and techniques described here can be implemented in acomputing system that includes a back end component (e.g., as a dataserver), or that includes a middleware component (e.g., an applicationserver), or that includes a front end component (e.g., a client computerhaving a graphical user interface or a Web browser through which a usercan interact with an implementation of the systems and techniquesdescribed here), or any combination of such back end, middleware, orfront end components. The components of the system can be interconnectedby any form or medium of digital data communication (e.g., acommunication network). Examples of communication networks include alocal area network (“LAN”), a wide area network (“WAN”), peer-to-peernetworks (having ad-hoc or static members), grid computinginfrastructures, and the Internet.

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other.

Although a few implementations have been described in detail above,other modifications are possible. Moreover, other mechanisms forperforming the systems and methods described in this document may beused. In addition, the logic flows depicted in the figures do notrequire the particular order shown, or sequential order, to achievedesirable results. Other steps may be provided, or steps may beeliminated, from the described flows, and other components may be addedto, or removed from, the described systems. Accordingly, otherimplementations are within the scope of the following claims.

1. A computer-implemented method, comprising: receiving spoken input ata computing device from a user of the computing device, the spoken inputincluding a carrier phrase and a subject to which the carrier phrase isdirected; providing at least a portion of the spoken input to a serversystem in audio form for speech-to-text conversion by the server system,the portion including the subject to which the carrier phrase isdirected; receiving from the server system instructions forautomatically performing an operation on the computing device, theoperation including an action defined by the carrier phrase usingparameters defined by the subject; and automatically performing theoperation on the computing device.
 2. The method of claim 1, wherein thecarrier phrase comprises an instruction to execute a navigationapplication, and the subject describes a geographical landmark.
 3. Themethod of claim 2, wherein the subject describes a geographical landmarkwithout naming the geographical landmark.
 4. The method of claim 2,wherein the carrier phrase is “navigate to.”
 5. The method of claim 2,wherein the operation comprises displaying turn-by-turn navigation in anapplication, wherein data for the turn-by-turn navigation is generatedby the server system for a navigation route from a current location ofthe computing device to a location described by the subject.
 6. Themethod of claim 1, wherein the prefix comprises an instruction to play amedia file, and the subject describes the media file to be played. 7.The method of claim 6, wherein the subject includes a name of content inthe media file.
 8. The method of claim 1, wherein the subject comprisesa search query and the prefix identifies operations to be conducted onsearch results for the search query.
 9. The method of claim 8, whereinthe operations to be conducted on the search results comprises parsing alanding page corresponding to a search result for a designated type ofinformation.
 10. The method of claim 9, wherein the designated type ofinformation comprises a location.
 11. A computer-implemented method,comprising: receiving, at a computer server system from a remotecomputing device, audio spoken by a user of the computing device, theaudio including a subject to which a spoken carrier phrase form the userwas directed; performing speech-to-text conversion on the audio by thecomputer server system; generating data for execution on the computingdevice to cause the computing device to automatically perform anoperation identified by the carrier phrase using the subject as aparameter of the operation; and transmitting the data to the computingdevice.
 12. The method of claim 11, further comprising receiving antextual identifier of the carrier phrase, wherein the computing deviceidentified the carrier phrase before providing the audio to the computerserver system.
 13. The method of claim 12, further comprising using theidentifier to select a grammar for performing the speech-to-textconversion.
 14. The method of claim 11, wherein the carrier phrasecomprises an instruction to execute a navigation application, and thesubject describes a geographical landmark.
 15. The method of claim 14,wherein the subject describes a geographical landmark without naming thegeographical landmark.
 16. The method of claim 14, wherein the carrierphrase is “navigate to.”
 17. The method of claim 14, wherein theoperation comprises displaying turn-by-turn navigation in anapplication, wherein data for the turn-by-turn navigation is generatedby the server system for a navigation route from a current location ofthe computing device to a location described by the subject.
 18. Themethod of claim 11, wherein the prefix comprises an instruction to playa media file, and the subject describes the media file to be played. 19.The method of claim 18, wherein the subject includes a name of contentin the media file.
 20. The method of claim 11, wherein the subjectcomprises a search query and the prefix identifies operations to beconducted on search results for the search query.
 21. The method ofclaim 20, wherein the operations to be conducted on the search resultscomprises parsing a landing page corresponding to a search result for adesignated type of information.
 22. The method of claim 21, wherein thedesignated type of information comprises a location.
 23. One or moretangible non-transient machine-readable media storing instructions that,when executed, perform actions comprising: receiving spoken input at acomputing device from a user of the computing device, the spoken inputincluding a carrier phrase and a subject to which the carrier phrase isdirected; providing at least a portion of the spoken input to a serversystem in audio form for speech-to-text conversion by the server system,the portion including the subject to which the carrier phrase isdirected; receiving from the server system instructions forautomatically performing an operation on the computing device, theoperation including an action defined by the carrier phrase usingparameters defined by the subject; and automatically performing theoperation on the computing device.
 24. The machine-readable media ofclaim 23, wherein the carrier phrase comprises an instruction to executea navigation application, and the subject describes a geographicallandmark.
 25. The machine-readable media of claim 23, wherein the prefixcomprises an instruction to play a media file, and the subject describesthe media file to be played.
 26. The machine-readable media of claim 23,wherein the subject comprises a search query and the prefix identifiesoperations to be conducted on search results for the search query.